Estimating Processor Load Using Frame Encoding Times

ABSTRACT

A method of estimating processor load at a device for transmitting a media stream, the device comprising an encoder and a processor capable of executing instructions for the encoder, the method comprising: encoding a first media frame and a second media frame at the encoder; determining a first time period between a first timestamp associated with the first media frame and a second timestamp associated with the second media frame; determining a second time period between a first completion time representing completion of the encoding of the first media frame and a second completion time representing completion of the encoding of the second media frame; and forming a measure of processor load in dependence on a difference between the first and second time periods.

BACKGROUND OF THE INVENTION

This invention relates to a method and device for estimating the load ofa processor.

Streaming of multimedia content over the internet has become anincreasingly common application in recent years. A wide range ofmultimedia applications, such as on-demand TV, live TV viewing, livevideo streaming, audio streaming, video conferencing, net meetings,video telephony, voice over internet protocol (VoIP) and many othersrely on end-to-end streaming solutions. There is increasing demand forthe streaming media to be played out at high quality.

High quality media usually requires a media source that can capture themedia at high resolutions and frame rates (amongst other parameters).For example, many video cameras are now able to capture video in highdefinition (HD), 4K or even 8K resolution. Even greater resolutions areenvisaged for future video applications. The raw data from such mediasources is not suitable for transmission over the internet due to thehigh data rates. For example, HD video has a sustained data rate of over100 MBps. The raw media data is therefore compressed before transmittingit over the internet.

Many codecs exist for compressing media data. Encoding the originalmedia data reduces its data rate whilst maintaining a certain level ofquality of the original source material. Encoding media requiressignificant processing capacity, especially when encoding high qualitymedia. Processing capacity on a device is usually shared with otherapplications and high processing demand from other applications can leadto an increase in the processor load and a reduction in the availablecapacity for encoding the media. This increase in processor load maylead to delays in encoding and transmitting a media stream. Such delayscan have undesirable effects such as delays in the playback of“real-time” media, packets not arriving at the playback device in atimely manner for playback, etc. There is, therefore, a need to estimatethe load on a processor so that, for example, a media stream can beadapted to negate any effects caused by the load on the processor.

SUMMARY OF THE INVENTION

According to a first aspect there is provided a method of estimatingprocessor load at a device for transmitting a media stream, the devicecomprising an encoder and a processor capable of executing instructionsfor the encoder, the method comprising: encoding a first media frame anda second media frame at the encoder; determining a first time periodbetween a first timestamp associated with the first media frame and asecond timestamp associated with the second media frame; determining asecond time period between a first completion time representingcompletion of the encoding of the first media frame and a secondcompletion time representing completion of the encoding of the secondmedia frame; and forming a measure of processor load in dependence on adifference between the first and second time periods.

The device may comprise a packetiser and the first completion time isdetermined in dependence on the first encoded frame being received atthe packetiser and the second completion time is determined independence on the second encoded frame being received at the packetiser.

The device may comprise a packetiser configured to packetise the encodedframes, and wherein the first completion time is determined independence on a first packet for the first encoded frame being outputfrom the packetiser; and the second completion time is determined independence on a first packet for the second encoded frame being outputfrom the packetiser.

The packetiser may be configured to packetise the encoded frames intoRTP packets.

The method may further comprise transmitting the measure to a device forreceiving the media stream.

The method may further comprise receiving a measure of delay in thenetwork; and adjusting a transmission bandwidth for the media stream independence on the measure of delay and the estimated processor load.

The method may further comprise, at the receiving device: forming ameasure of delay in the network; receiving the measure of processor loadvia the network; and adjusting the measure of delay in dependence on themeasure of processor load.

The measure of network delay may be formed by: determining a third timeperiod between receiving a first-received packet for the first mediaframe and receiving a first-received packet for the second media frameat a receiving device, wherein the packet for the first media framecomprises the first timestamp and the packet for the second media framecomprises the second timestamp; determining a fourth time period betweenthe first timestamp and the second timestamp; and forming the measure ofdelay in dependence on the difference between the third and fourth timeperiods.

The measure of network delay may be formed by: determining a third timeperiod between receiving the first media frame and receiving the secondmedia frame, wherein the first media frame comprises the first timestampand the second media frame comprises the second timestamp; determining afourth time period between the first timestamp and the second timestamp;and forming the measure of delay in dependence on the difference betweenthe third and fourth time periods.

The method may further comprise determining that the processor isoverloaded if the second time period is greater than the first timeperiod.

The method may further comprise: in response to determining that theprocessor is overloaded, changing parameters for encoding the mediaframes so as to cause a reduction in the quality of the frames to beoutputted by the encoder.

According to a second aspect there is provided a method of identifying anetwork condition between a pair of network devices, wherein one of thedevices comprises a processor capable of executing instructions forforming a media stream for transmission over the network, the methodcomprising: monitoring a measure of delay in receiving media over thenetwork; monitoring a measure of load on the processor; and identifyinga network condition in dependence on a change in the measure of delayand the measure of load on the processor.

The identifying step may further comprise identifying congestion in thenetwork if the change in the measure of delay indicates an increase innetwork delay and the measure of load indicates that the processor isnot overloaded.

The identifying step may comprise identifying that the network is notcongested if the change in the measure of delay indicates an increase innetwork delay and the measure of load indicates that the processor isoverloaded.

The step of monitoring a measure of delay may comprise: determining afirst time period between receiving a first-received packet for aninitial media frame and receiving a first-received packet for asubsequent media frame, wherein each received packet comprises atimestamp; determining a second time period between the timestamp of thepacket for the initial media frame and the timestamp of the packet forthe subsequent media frame; and forming the measure of delay independence on the difference between the first and second time periods.

The packets may be RTP packets. The measure of delay may be determinedin dependence on RTP timestamps.

The step of monitoring a measure of delay may comprise: determining afirst time period between receiving an initial media frame and receivinga subsequent media frame, wherein each received frame comprises atimestamp; determining a second time period between the timestamp of theinitial media frame and the timestamp of the subsequent media frame; andforming the measure of delay in dependence on the difference between thefirst and second time periods.

According to a third aspect there is provided a device for transmittinga media stream, the device comprising: an encoder configured to encodinga first media frame and a second media frame; a processor capable ofexecuting instructions for the encoder; and a controller configured to:determine a first time period between a first timestamp associated withthe first media frame and a second timestamp associated with the secondmedia frame; determine a second time period between a first completiontime representing completion of the encoding of the first media frameand a second completion time representing completion of the encoding ofthe second media frame; and form a measure of processor load independence on a difference between the first and second time periods.

The device may comprise a packetiser configured to packetise the encodedframes, wherein the controller is configured to determine the firstcompletion time in dependence on the first encoded frame being receivedat the packetiser and determine the second completion time in dependenceon the second encoded frame being received at the packetiser.

The device may further comprising a packetiser configured to packetisethe encoded frames, wherein the controller is configured to determinethe first completion time in dependence on a first packet for the firstencoded frame being output from the packetiser and determine the secondcompletion time in dependence on a first packet for the second encodedframe being output from the packetiser.

The packetiser may be configured to packetise the encoded frames intoRTP packets.

The controller may be configured to determine that the processor isoverloaded if the second time period is greater than the first timeperiod.

In response to determining that the processor is overloaded, thecontroller may be configured to change parameters for encoding the mediaframes so as to cause a reduction in the quality of the frames to beoutputted by the encoder.

The device may further comprise a transceiver configured to transmit themeasure to a device for receiving the media stream.

The transceiver may be configured to receive a measure of delay in thenetwork; and the controller is configured to adjust a transmissionbandwidth for the media stream in dependence on the measure of delay andthe estimated processor load.

According to a fourth aspect there is provided a device for receiving amedia stream via a network, the device comprising: a controllerconfigured to form a measure of delay in the network; a transceiverconfigured to receive a measure of processor load from a devicetransmitting the media stream; and a controller configured to adjust themeasure of delay in dependence on the measure of processor load.

The measure of network delay may be formed by: determining a first timeperiod between receiving a first-received packet for a first media frameand receiving a first-received packet for a second media frame at thedevice, wherein the packet for the first media frame comprises a firsttimestamp and the packet for the second media frame comprises a secondtimestamp; determining a second time period between the first timestampand the second timestamp; and forming the measure of delay in dependenceon the difference between the first and second time periods.

The measure of network delay may be formed by: determining a first timeperiod between receiving a first media frame and receiving a secondmedia frame, wherein the first media frame comprises a first timestampand the second media frame comprises a second timestamp; determining asecond time period between the first timestamp and the second timestamp;and forming the measure of delay in dependence on the difference betweenthe first and second time periods.

According to a fifth aspect there is provided a device for identifying anetwork condition, the device comprising a controller configured to:monitor a measure of delay in receiving a media stream over the network;monitor a measure of load on a processor capable of executinginstructions for forming the media stream; and identify a networkcondition in dependence on a change in the measure of delay and themeasure of load on the processor.

The controller may be configured to identify congestion in the networkif the change in the measure of delay indicates an increase in networkdelay and the measure of load indicates that the processor is notoverloaded.

The controller may be configured to identify that the network is notcongested if the change in the measure of delay indicates an increase innetwork delay and the measure of load indicates that the processor isoverloaded.

The controller may be configured to monitor the measure of delay by:determining a first time period between receiving a first-receivedpacket for an initial media frame and receiving a first-received packetfor a subsequent media frame, wherein each received packet comprises atimestamp; determining a second time period between the timestamp of thepacket for the initial media frame and the timestamp of the packet forthe subsequent media frame; and forming the measure of delay independence on the difference between the first and second time periods.

The packets may be RTP packets. The measure of delay may be determinedin dependence on RTP timestamps.

The controller may be configured to monitor the measure of delay by:determining a first time period between receiving an initial media frameand receiving a subsequent media frame, wherein each received framecomprises a timestamp; determining a second time period between thetimestamp of the initial media frame and the timestamp of the subsequentmedia frame; and forming the measure of delay in dependence on thedifference between the first and second time periods.

According to a sixth aspect there is provided machine readable code forimplementing the method described above.

According to a seventh aspect there is provided a machine readablenon-transitory storage medium having encoded thereon machine readablecode for implementing a method as described above.

DESCRIPTION OF THE DRAWINGS

The present invention will now be described by way of example withreference to the accompanying drawings. In the drawings:

FIG. 1 shows an example of a transmitting device and a receiving device;

FIGS. 2a-2d illustrate frame reception timings under various processorloads and encoder parameters;

FIGS. 3a-3d illustrate frame reception timings under various networkconditions; and

FIG. 4 is a flow diagram illustrating an example of estimating processorload.

DETAILED DESCRIPTION OF THE DRAWINGS

The following description is presented to enable any person skilled inthe art to make and use the invention, and is provided in the context ofa particular application. Various modifications to the disclosedembodiments will be readily apparent to those skilled in the art.

The general principles defined herein may be applied to otherembodiments and applications without departing from the spirit and scopeof the present invention. Thus, the present invention is not intended tobe limited to the embodiments shown, but is to be accorded the widestscope consistent with the principles and features disclosed herein.

FIG. 1 depicts a transmitting device 10, which may be any suitabledevice that is capable of generating packet based data such as acomputer, smartphone, videophone, etc. The transmitting device 10comprises a transceiver 11 for connection to a communications network 12such as the internet or other packet based networks. The transmittingdevice 10 can transmit and/or receive packets to and/or from thecommunications network 12 via the transceiver 11.

The transmitting device 10 comprises a media source 13. The media source13 may be, for example, a camera, microphone, a storage device (e.g.flash memory, hard disk), a removable storage device (e.g. memory card,CD), a networked storage device (e.g. network drive or the cloud), aninternet media provider (e.g. a streaming service), radio (e.g. DAB),etc. The media source 13 may be comprised within the transmitting device10 or connected to the transmitting device 10 via a wired or wirelessinterface such as HDMI, USB, analogue line-in, I2S, S/PDIF, Bluetooth,Ethernet, Wi-Fi, DAB, etc.

The transmitting device 10 comprises an encoder 14 for encoding mediadata (e.g. video and/or audio data) from the media source 13. The mediafrom the media source 13 may be provided as media frames. The encoder 14may encode the media frames according to a coding standard such as ITU-TRecommendation H.264 or ISO/IEC International Standard 14496-10 (bothalso known as Advanced Video Coding (AVC)), MPEG-DASH, HTTP LiveStreaming or any other suitable codec.

The transmitting device 10 may comprise a packetiser 15 which receivesthe encoded media frames from the encoder 14 and packetises the framesinto a sequence of packets for transmission over the network 12 via thetransceiver 11. Each frame may be packetised into one or more packets.The packetiser 15 may packetise the media in accordance with a Real-timeTransport Protocol (RTP) standard. Other standardised packet formats maybe used. The packetiser 15 provides the packets to the transceiver 11for transmission over the network 12 to receiving device 20.

The receiving device 20 comprises a transceiver 21 for receiving packetsfrom the network 12. The packets may be provided to a buffer 22, whichmay be a jitter buffer that is capable of ordering the packets accordingto a playout sequence of the media data in the packets. This sequencemay be indicated by a sequence number or timestamp contained in eachpacket. A decoder 23 decodes the packets in the order provided to it bythe buffer 22 to form a media stream. The decoder 23 decodes packetsaccording to the codec used by the encoder 13. A media consumer 24receives the decoded media stream for playback. The media consumer 24may be, for example, an audio and/or video player.

The transmitting device 10 comprises a processor 16 (which may be, forexample, a CPU or GPU). The processor 16 executes instructions forapplications running at the transmitting device 10. These instructionsmay come from the encoder 14 for encoding the media frames received fromthe media source 13, from the packetiser 15 for packetising the encodedmedia frames or from any other application (not shown) that may berunning at the transmitting device 10.

As mentioned above, on occasions the processor 16 may become overloaded.Generally, the processor 16 could be considered to be overloaded whenother applications are utilising the majority of the processor'sresources such that it affects the operation of the encoder 13.Overloading of the processor 16 may be caused by spikes in the demand,thermal throttling, rogue processes, or other reasons. Overloading ofthe processor 16 may lead to problems with the media stream being sentto the receiving device 20. Thus, it is desirable to determine the loadon the processor 16 so that the amount of work required to be performedby the processor 16 can be appropriately adapted.

The transmission device 10 may comprise a controller 17, which iscapable of estimating the load on the processor 16 and, based on theestimated load, causing the amount of work required to be carried out bythe processor 16 to be adapted. For example, if the controller 17determines that the processor 16 is overloaded, it may cause the encoder14 to use a codec that is less demanding on the processor 16.

The controller 17 may estimate the load on the processor 16 by comparingthe difference between the rate at which frames are provided to theencoder 14 and the rate at which the encoded frames are output from theencoder 14. As mentioned above, encoding media frames generally requiressignificant processing capacity and so measuring the amount of time theencoder 14 takes to encode a frame provides an indication of the load onthe processor 16.

Each frame from the media source 13 is associated with a timestamp. Thetimestamp may indicate the time that the media from the media source 13was sampled (i.e. a sampling instant) according to a clock (not shown),which may be a wallclock that is synchronised at the transmitting andreceiving devices 10 and 20. The rate at which frames are provided tothe encoder 14 from the media source 13 may be determined from thetimestamps of those frames. The rate at which the encoded frames areoutput from the encoder 14 may be determined from the time that eachencoded frame is received at one of the data processing units in themedia processing pipeline after the encoder 14 (e.g. the packetiser 15or transceiver 11 or any other suitable unit that may be present betweenthe encoder 14 and transceiver 11). The input and output frame rates arecompared to form to estimate the load on the processor. For example, ifthe output frame rate is less than the input frame rate, then this mayindicate that the processor 16 is overloaded. The controller 17 may thentake appropriate action to reduce the demand on the processor 16.

Preferably, the controller 17 forms a load measure for estimating theload on the processor 16. The load measure may be based on a timedifference between receiving each newly encoded frame from the encoder14 and the time of receiving a reference encoded frame from the encoder14 and comparing that time difference against the difference in timebetween the timestamps of those frames. Equation 1 is an example of howthe comparison could be made:

Load measure=(CurrentFramePk−RefFramePk)−(CurrentFrameTS−RefFrameTS)  (1)

Where CurrentFramePk is the time when the encoded current frame isreceived at the packetiser 15, RefFramePk is the time when an encodedreference frame is received at packetiser 15, CurrentFrameTS is thetimestamp for the encoded current frame and RefFrameTS is the timestampfor the encoded reference frame. The timestamp for a frame may becomprised within that frame.

In this example and the examples below, the time at which the encodedframes are received at the packetiser 15 is determined. However, asmentioned above, the point at which the time for receiving an encodedframe is determined may be at any point after the encoding of that framehas been completed, which will usually be at some point after theencoder 14. For example, the time at which the first packet for anencoded frame is received by the transceiver 11 from the packetiser 15may be used to determine CurrentFrameRx and RefFrameRx. The amount ofprocessing resources used by the packetiser 15 for packetisation isrelatively negligible compared to encoding media and so there is nosignificant difference between determining the time for receivingencoded frames prior to the packetiser 15 or receiving packets for thoseframes after the packetiser 15.

Equation 2 is an example of how the congestion measure can be determinedin a system which uses an RTP packets to send media data:

Loadmeasure=(CurFrameWALLTS−RefFrameWALLTS)−[(CurFrameRTPTS−RefFrameRTPTS)/90]  (2)

Where CurFrameWALLTS is the time in milliseconds when the current frameis received at the packetiser 15, RefFrameWALLTS is the timemilliseconds when the reference frame is received at the packetiser 15,CurFrameRTPTS is the timestamp comprised in the current frame andRefFrameRTPTS is the timestamp comprised in the reference frame. TheCurFrameRTPTS and RefFrameRTPTS timestamps may be determined and appliedby the encoder based on frame rate information that is provided by themedia source 13. Dividing by 90 converts the RTP time units of thetimestamp into milliseconds.

The reference frame is considered to be a frame that has been encoded inthe shortest time possible according to the best available knowledge.Initially, the first frame of the media stream is chosen to be thereference frame. Subsequently, the reference frame is updated whenever aframe is encoded faster than the previous reference frame.

FIGS. 2a-2d illustrate various scenarios which show how the load measureis determined and how it is used to estimate the load on the processor16.

FIG. 2a illustrates a scenario where the processor 16 has a low load andis operating under normal conditions. Each arrow represents a mediaframe that has been encoded by the encoder 14. The RTP time stamp(RTPTS) for that frame is indicated above the arrows. The time when eachencoded frame is received at the packetiser 15 is indicated by the timebelow the arrows (in milliseconds). This time is shown as a timerelative to the first frame, but it could also be an absolute time. Inthis case the first frame is selected to be the reference frame. Usingequation 2 for frames with an RTP timestamp, the load measure for eachframe is calculated as follows:

1^(st) frame:

Load measure=(0−0)−((10000−10000)/90)=0

2^(nd) frame:

Load measure=(100−0)−((19000−10000)/90)=(100−100)=0

3^(rd) frame:

Load measure=(200−0)−((28000−10000)/90)=(200−200)=0

4^(th) frame:

Load measure=(300−0)−((37000−10000)/90)=(300−300)=0

11th frame:

Load measure=(10000−0)−((910000−10000)/90)=(10000−10000)=0

21st frame:

Load measure=(20000−0)−((1810000−10000)/90)=(20000−20000)=0

In this scenario, the load measure is a constant value of zero from the1^(st) frame to the 21^(st) frame. This indicates that the frames arebeing encoded by the encoder 14 at the best known rate and so theprocessor is not overloaded and operating satisfactorily for thepurposes of transmitting the media stream.

FIG. 2b illustrates a scenario where the processor 16 is overloaded.Using equation 2, the load measure for each frame is calculated asfollows:

1st frame:

Load measure=(0−0)−((10000−10000)/90)=0

2nd frame:

Load measure=(200−0)−((19000−10000)/90)=(200−100)=100

3rd frame:

Load measure=(400−0)−((28000−10000)/90)=(400−200)=200

4^(th) frame:

Load measure=(1000−0)−((37000−10000)/90)=(1000−300)=700

11th frame:

Load measure=(20000−0)−((910000−10000)/90)=(20000−10000)=10000

21st frame:

Congestion measure=(40000−0)−((1810000−10000)/90)=(40000−20000)=20000

This scenario differs from the FIG. 2a scenario as the frames for the2^(nd) to 21^(st) frames are being received at the packetiser 15 in anincreasingly delayed manner. When the processor 16 is overloaded, thereis less processing capacity available for the encoder 14 to encode theincoming frames. Thus, it takes a longer amount of time to complete theencoding of a frame which leads to a delay in receiving the encodedframe at the packetiser 15. The slowdown at the encoder 14 can lead to abuild up for frames at the input buffer of the encoder 14. This leads tothe frames being increasingly delayed and so the load measure valueincreases for each subsequent frame. Thus, the increasing load measureindicates that the processor 16 is overloaded.

FIG. 2c illustrates a scenario where there is an increase in the time ittakes the encoder 14 to encode a frame, which may be due to, forexample, a change in the pixel resolution required for frames outputtedby the encoder 14 or any other change that may require greaterprocessing resources for encoding a frame. This change may have beeninitiated due to another control algorithm such as a network bandwidthadaptation algorithm which may adapt the transmission bitrate inresponse to changes in the available network bandwidth. In the exampleof FIG. 2c , the increase in encoding time occurs between the second andthird frame due to an increase in the pixel resolution of the frames (asindicated in the figure). Using equation 2, the load measure for eachframe is calculated as follows:

1st frame:

Load measure=(0−0)−((10000−10000)/90)=0

2nd frame:

Load measure=(100−0)−((19000−10000)/90)=(100−100)=0

3rd frame:

Load measure=(250−0)−((28000−10000)/90)=(250−200)−50

4th frame:

Load measure=(350−0)−((37000−10000)/90)=(350−300)=50

11th frame:

Load measure=(10050−0)−((910000−10000)/90)=(10050−10000)=50

21st frame:

Load measure=(20050−0)−((1810000−10000)/90)=(20050−20000)=50

In this scenario the load measure increases for the third frame andremains constant at that increased value for the subsequent frames. Thisindicates that the processor 16 is not overloaded and operatingsatisfactorily as the rate at which frames are inputted and outputted bythe encoder 14 are the same. The increase in delay indicates, forexample, that there has been an increase in the encoding time for eachframe after the second frame which may be due to a change in the pixelresolution of the frames encoded by the encoder 14. Preferably, whenthis scenario is determined by the quality controller 15, the constantvalue (50, in this example), is subtracted from the calculation of theload measure until the reference frame has been updated.

FIG. 2d illustrates a scenario where there is a decrease in the time ittakes the encoder 14 to encode a frame, which may be due to, forexample, a change in the pixel resolution used required for framesoutputted by the encoder 14 or any other change that may require lessprocessing resources for encoding a frame. In this example, the decreasein encoding time occurs between the third and fourth frame due to adecrease in the pixel resolution of the frames (as indicated in thefigure). Using equation 2, the load measure for each frame is calculatedas follows:

1st frame:

Load measure=(0−0)−((10000−10000)/90)=0

2nd frame:

Load measure=(100−0)−((19000−10000)/90)=(100−100)=0

3rd frame:

Load measure=(200−0)−((28000−10000)/90)=(200−200)=0

4th frame:

Load measure=(250−0)−((37000−10000)/90)=(250−300)=−50

Update the reference frame, RefFrame=FourthFrame

11th frame:

Load measure=(9950−250)−((910000−37000)/90)=(9700−9700)=0

21st frame:

Load measure=(19950−250)−((1810000−37000)/90)=(19700−19700)=0

In this scenario the load measure decreases for the fourth frame. Thedecrease indicates, for example, that there has been a change in thepixel resolution from the fourth frame which leads to a shorter amountof time being required to encode frames at the encoder 14. In this case,the fourth frame is considered to be the frame that has been encodedquicker than the first frame and so the reference frame is updated fromthe first frame to the fourth frame for calculating the load measure forsubsequent frames.

The controller 17 may monitor changes in the load measure to determinethe load on the processor 16, as described by the scenarios above. Basedon the determined load on the processor the controller 17 may adapt theencoding parameters for the encoder 14 in order to reduce the demand onthe processor so that frames can be transmitted in a timely manner. Forexample, if it is determined that the processor 16 is overloaded, thecontroller 17 may cause the encoder 14 to decrease the quality (e.g. bydecreasing the pixel resolution) of the frames to be outputted by theencoder 14 so that the encoding process is less resource intensive. Thiswill reduce the processing required per frame and so reduce the amountof time required to encode each frame. Other encoder parameters may beadapted by the controller 15 in response to a processor overload (suchas output frame rate, bitrate, number of colours, frame types, etc.).The controller 17 may be configured to cause other applications on thetransmitting device 10 to reduce demand on the processor 16 if it isoverloaded. This way, there will be greater processing resourcesavailable to the encoder 14 and so the quality of the media stream iscapable of being maintained.

Determination of the load on the processor may be used in combinationwith a measure of network delay to accurately determine the condition ofnetwork 12. The condition of the network 12 may change (e.g. it maybecome congested), which can lead to a delay in receiving packets at thereceiving device 20. This delay may lead to complete frames beingreceived too late for them to be played out on time by the mediaconsumer 24. Thus, it is desirable to determine if the condition of thenetwork 12 has changed and how it has changed so that the transmittingdevice 10 can appropriately adapt its transmission in order tocompensate for the change.

The receiving device 20 comprises a controller 25 for identifyingchanges in the network 12. The controller 25 is capable of determiningwhen packets are received by the receiving device 20. The time that apacket is received may be derived from the wallclock or an internalclock (not shown), which may not necessarily be synchronised with thewallclock. The controller 25 is also capable of determining the timeindicated by the timestamp comprised in each received packet. Thecontroller 25 is capable of identifying changes in the condition of thenetwork by comparing the times when packets are received at thereceiving device 20 with the times indicated by the timestamps of thosepackets (as discussed in further detail below). The controller 25 maysend an indication of the identified change to the transmitting device10 so that it can appropriately adjust its transmission parameters. Thecontroller 25 may also adjust some of its reception parameters (e.g.target jitter buffer size) in response to some network conditions.

Preferably, a congestion measure (described below) is used as themeasure of network delay. However, other measures of network delay maybe used, such as:

-   -   packet round-trip time.    -   measuring inter-arrival times between complete frames.    -   measuring the time period between timestamps of frames or        packets received over a fixed period (e.g. one second) and        comparing the time period with the fixed period.    -   measuring the spread between pairs of packets of varying sizes        and estimate the available bandwidth as a function of the time        spread between packets in a pair, and their size. Depending upon        the available bandwidth, the packet pairs spread in time as they        traverse the network elements.

The quality controller 25 may use the congestion measure to identifynetwork conditions between the transmitting and receiving devices 10 and20 such as congestion, route changes, etc. The congestion measure isdetermined using the inter-arrival times between each newly receivedframe and a reference frame. Each frame may be made up from one or morepackets having the same timestamp. The inter-arrival times betweenframes may be determined by comparing the arrival of a complete frame orthe arrival of the first-received packet for each frame. Thefirst-received packet for each frame may not necessarily be the firstpacket that has been packetised or transmitted by the transmittingdevice 10 because of packet loss, varying network paths, etc. in thenetwork 12.

The inter-arrival times of a newly received frame and a reference frameis compared against the difference in time between the timestamps ofthose frames. This comparison is used to form the congestion measure.Equation 3 is an example of how the comparison could be made:

Congestionmeasure=(CurrentFrameRx−RefFrameRx)−(CurrentFrameTS−RefFrameTS)   (3)

Where CurrentFrameRx is the time when when the first packet is receivedfor the current frame at the receiving device 20, RefFrameRX is the timewhen the first packet is received for the reference frame at thereceiving device 20, CurrentFrameTS is the timestamp for the currentframe and RefFrameTS is the timestamp in the reference frame.Alternatively, CurrentFrameRx and RefFrameRX may be the times when thecurrent frame is received and when the reference frame is receivedrespectively.

Equation 4 is an example of how the congestion measure can be determinedin a system which uses RTP packets to send media data:

Congestionmeasure=(CurrentFrameRx−RefFrameRx)−[(CurrentFrameTS−RefFrameTS)/90]  (4)

Where CurrentFrameRx is the time in milliseconds when the first RTPpacket is received for the current frame at the receiving device 20,RefFrameRX is the time milliseconds when the first RTP packet isreceived for the reference frame at the receiving device 20,CurrentFrameTS is the timestamp comprised the RTP header of the firstRTP packet for the current frame and RefFrameTS is the timestampcomprised in the RTP header of the first RTP packet for the referenceframe. Alternatively, CurrentFrameRx is the time in milliseconds whenthe current frame is received, RefFrameRX is the time milliseconds whenthe reference frame is received and CurrentFrameTS and RefFrameTS arethe timestamps comprised in the current and reference framesrespectively. Dividing by 90 converts the RTP time units intomilliseconds.

If the congestion measure is zero or within some threshold that is closeto zero, then it is considered that packets received from the networkare “on-time” and so the network is operating satisfactorily for thepurposes of the media stream. The reference frame is considered to be aframe that has arrived on-time according to the best availableknowledge. Initially, the first frame of the media stream is chosen tobe the reference frame. Subsequently, the reference frame is updatedwhenever a frame arrives on-time or within some threshold.

FIGS. 3a-3d illustrate various scenarios which show how the congestionmeasure can be used to determine the condition of a network.

FIG. 3a illustrates a scenario where the network is operating undernormal conditions. Each arrow represents a media frame sent using theRTP protocol. The RTP time stamp (RTPTS) for that frame is indicatedabove the arrows. The time when then the first packet for each of theframes is received is indicated by the time below the arrows (inmilliseconds). This time is shown as a time relative to the first frame,but could also be an absolute time. In this case the first frame isselected to be the reference frame. Using equation 4 for RTP packets,the congestion measure for each frame is calculated as follows:

1st frame:

Congestion measure=(0−0)−((10000−10000)/90)=0

2nd frame:

Congestion measure=(100−0)−((19000−10000)/90)=(100−100)=0

3rd frame:

Congestion measure=(200−0)−((28000−10000)/90)=(200−200)=0

4th frame:

Congestion measure=(300−0)−((37000−10000)/90)=(300−300)=0

11th frame:

Congestion measure=(10000−0)−((910000−10000)/90)=(10000−10000)=0

21st frame:

Congestion measure=(20000−0)−((1810000−10000)/90)=(20000−20000)=0

In this scenario, the congestion measure is a constant value of zerofrom the 1st frame to the 21st frame. This indicates that the frames arereaching the receiving device 20 on-time and so the network is operatingsatisfactorily for the purposes of the media stream.

FIG. 3b illustrates a scenario where there is congestion in the network.Using equation 4, the congestion measure for each frame is calculated asfollows:

1st frame:

Congestion measure=(0−0)−((10000−10000)/90)=0

2nd frame:

Congestion measure=(200−0)−((19000−10000)/90)=(200−100)=100

3rd frame:

Congestion measure=(400−0)−((28000−10000)/90)=(400−200)=200

4^(th) frame:

Load measure=(1000−0)−((37000−10000)/90)=(1000−300)=700

11th frame:

Congestion measure=(20000−0)−((910000−10000)/90)=(20000−10000)=10000

21st frame:

Congestion measure=(40000−0)−((1810000−10000)/90)=(40000−20000)=20000

This scenario differs from the FIG. 3a scenario as the first-receivedpacket for the 2nd to 21st frames are being received in an increasinglydelayed manner. During congestion, the congestion measure for each frameis proportional to the mismatch between the sending rate and thebottleneck bandwidth. At other times, the congestion measure is close tozero or below some threshold indicating no congestion. The manner inwhich the congestion measure increases depends on the sending bandwidth,choke bandwidth (or throttling) and network usage.

FIG. 3c illustrates a scenario where there is a single increase innetwork delay, which occurs between the second and third frame (asindicated in the figure). Using equation 4, the congestion measure foreach frame is calculated as follows:

1st frame:

Congestion measure=(0−0)−((10000−10000)/90)=0

2nd frame:

Congestion measure=(100−0)−((19000−10000)/90)=(100−100)=0

3rd frame:

Congestion measure=(250−0)−((28000−1000)/90)=(250−200)=50

4th frame:

Congestion measure=(350−0)−((37000−10000)/90)=(350−300)=50

11th frame:

Congestion measure=(10050−0)−((910000−10000)/90)=(10050−10000)=50

21st frame:

Congestion measure=(20050−0)−((1810000−10000)/90)=(20050−20000)=50

In this scenario the congestion measure increases for the third frameand remains at that increased value for the subsequent frames. As thedelay is constant (rather than increasing, as is the case for FIG. 3b ),this indicates that the delay is not due to congestion. This increase indelay indicates, for example, that there has been a change in thenetwork path (as depicted by the route change arrow in FIG. 3c ) whichleads to a longer amount of time for packets to be transported from thetransmitting device 10 to the receiving device 20. Preferably, when thisscenario is determined by the quality controller 25, the constant value(50, in this example), is subtracted from the calculation of thecongestion measure until the reference frame has been updated.

FIG. 3d illustrates a scenario where there is a single decrease innetwork delay, which occurs between the third and fourth frames, asindicated in the figure. Using equation 4, the congestion measure foreach frame is calculated as follows:

1st frame:

Congestion measure=(0−0)−((10000−10000)/90)=0

2nd frame:

Congestion measure=(100−0)−((19000−10000)/90)=(100−100)=0

3rd frame:

Congestion measure=(200−0)−((28000−10000)/90)=(200−200)=0

4th frame:

Congestion measure=(250−0)−((37000−10000)/90)=(250−300)=−50

Update the reference frame, RefFrame=FourthFrame

11th frame:

Congestion measure=(9950−250)−((910000−37000)/90)=(9700−9700)=0

21st frame:

Congestion measure=(19950−250)−((1810000−37000)/90)=(19700−19700)=0

In this scenario the congestion measure decreases for the fourth frame.The decrease indicates, for example, that there has been a change in thenetwork path (as depicted by the route change arrow in FIG. 3d ) whichleads to a shorter amount of time for packets to be transported from thetransmitting device to the receiving device. In this case, the fourthframe is considered to be received on-time and so the reference frame isupdated from the first frame to the fourth frame for calculating thecongestion measure for subsequent frames.

Comparing the scenarios described in respect of the network congestionmeasure (FIG. 3a-3d ) and the scenarios described in respect of theprocessor load measure (FIGS. 2a-2d ), it can be seen that certainconditions at the transmitting device 10 may present itself as a networkcondition to the receiving device 20. For example, when the processor 16at the transmitting device 10 becomes overloaded (as described above inrelation to FIG. 2b ), there is an increasing delay in encoding theframes. This increasing delay leads to a delayed transmission ofpackets, which in turn leads to a delay in receiving the packets at thereceiving device 20. This leads to the receiving device 20 observing anincreasing congestion measure being even though the network is behavingnormally. Thus a processor overload at the transmitting device 10presents itself as network congestion to the controller 25 at thereceiving device 20. Similarly, the changes in pixel resolutiondescribed in relation to FIGS. 2c and 2d may present themselves aschanges in the network path described in relation to FIGS. 3c and 3 d.

The network congestion measure may be adjusted in dependence on theprocessor load measure to avoid any changes at the transmitting device10 presenting themselves as changes in the network 12. This may beachieved by subtracting the processor load measure for each frame fromthe network congestion measure for each corresponding frame. Thecontroller 17 at the transmitting device 10 may transmit an indicationof the processor load measure for each frame to the receiving device 20.The controller 25 at the receiving device 20 may determine thecongestion measure for each frame and then subtract the received loadmeasure for the corresponding frames and use the resulting values todetermine the condition of the network.

Alternatively, the controller 25 of the receiving device 20 may send thecongestion measure for each frame to the transmitting device 10 so thatcontroller 17 can determine the condition of the network 12. Thecontroller 17 may temporally store the processor load measure for eachframe the transmitting device 10 transmits and then subtract the loadmeasure from the congestion measure it receives for each frame. Theresulting value may then be used to determine the condition of thenetwork. The transmitting device 10 may then adjust its mediatransmission in dependence on the determined network condition. Forexample, if it is determined that the network is congested, the qualitycontroller 15 may, in response, cause the transmitting device 10 toreduce its transmission bandwidth. This may be achieved by, for example,reducing the quality of the media that is to be transmitted, increasingpacket size (which reduces the packetisation overhead), reducing errorcorrection redundancy, etc. Preferably, the transmission bandwidth willbe reduced to be lower than the congested network bandwidth so thatfurther congestion can be avoided. This will ensure that transmittedmedia packets arrive at the receiving device in a timely manner forplayback.

The load measure provides media streaming applications with a simple andresource efficient method of monitoring the load of a processor. Thisnegates the need for any separate processes or applications forspecifically monitoring and reporting the processor load to the mediastreaming application. The load measure may additionally be used incombination with a congestion measure to provide an accurate measure ofnetwork delay, which can then be used to reliably determine changes inthe condition of a network and the causes of those changes. Atransmitting device may appropriately adjust its encoding and/ortransmission parameters to compensate for changes in the processor loadand/or changes in the network to maintain a level of quality for playingback the media stream that a receiving device.

FIG. 4 is a flow diagram illustrating an example of how the load onprocessor 16 may be estimated.

At step 401, the encoder 14 receives a frame for encoding from the mediasource 13. At step 402, the encoder encodes the frame and applies atimestamp to the frame. The timestamp may be based on the frame rate ofthe media outputted by the media source 13. At step 403, the time atwhich encoding of the frame has been completed (e.g. the when theencoded frame is received by the packetiser 15) is stored.

At step 404, it is checked if a reference frame has been set. If so,then the process proceeds to step 406. If not, then the frame is set asthe reference frame at step 405 and the process returns to step 401 forthe next frame received by the encoder 14.

At step 406, the applied timestamp is compared to a timestamp of thereference frame to determine a first time period. At step 407, thestored completion time (stored at step 403) is compared to a time whenencoding of the reference frame was completed to determine a second timeperiod. At step 408, the difference between the first and second timeperiods is determined to provide a measure of processor load.

The process of FIG. 4 may be performed for every frame received from themedia source (as indicated by the process looping back to step 401).

The transmitting and receiving devices configured in accordance with theexamples described herein could be embodied in hardware, software or anysuitable combination of hardware and software. The transmitting devicesmay have the same capabilities as the receiving devices and vice versa.The devices as described herein could comprise, for example, softwarefor execution at one or more processors (such as at a CPU and/or GPU),and/or one or more dedicated processors (such as ASICs), and/or one ormore programmable processors (such as FPGAs) suitably programmed so asto provide functionalities of the devices, and/or heterogeneousprocessors comprising one or more dedicated, programmable and generalpurpose processing functionalities. The devices described herein cancomprise one or more processors and one or more memories having programcode stored thereon, the processors and the memories being such as to,in combination, provide the claimed devices and/or perform the claimedmethods.

Data processing units described herein (e.g. encoder, controller andpacketizer) need not be provided as discrete units and representfunctionalities that could (a) be combined in any manner, and (b)themselves comprise one or more data processing entities. Dataprocessing units could be provided by any suitable hardware or softwarefunctionalities, or combinations of hardware and softwarefunctionalities.

The term software as used herein includes executable code for processors(e.g. CPUs and/or GPUs), firmware, bytecode, programming language codesuch as C or OpenCL, and modules for reconfigurable logic devices suchas FPGAs. Machine-readable code includes software and code for defininghardware, such as register transfer level (RTL) code as might begenerated in Verilog or VHDL.

Any one or more of the methods described herein could be performed byone or more physical processing units executing program code that causesthe unit(s) to perform the methods. The or each physical processing unitcould be any suitable processor, such as a CPU or GPU (or a corethereof), or fixed function or programmable hardware. The program codecould be stored in non-transitory form at a machine readable medium suchas an integrated circuit memory, or optical or magnetic storage. Amachine readable medium might comprise several memories, such as on-chipmemories, computer working memories, and non-volatile storage devices.

The applicant hereby discloses in isolation each individual featuredescribed herein and any combination of two or more such features, tothe extent that such features or combinations are capable of beingcarried out based on the present specification as a whole in the lightof the common general knowledge of a person skilled in the art,irrespective of whether such features or combinations of features solveany problems disclosed herein, and without limitation to the scope ofthe claims. The applicant indicates that aspects of the presentinvention may consist of any such individual feature or combination offeatures. In view of the foregoing description it will be evident to aperson skilled in the art that various modifications may be made withinthe scope of the invention.

1. A method of estimating processor load at a device for transmitting amedia stream, the device comprising an encoder and a processor capableof executing instructions for the encoder, the method comprising:encoding a first media frame and a second media frame at the encoder;determining a first time period between a first timestamp associatedwith the first media frame and a second timestamp associated with thesecond media frame; determining a second time period between a firstcompletion time representing completion of the encoding of the firstmedia frame and a second completion time representing completion of theencoding of the second media frame; and forming a measure of processorload in dependence on a difference between the first and second timeperiods.
 2. A method as claimed in claim 1, wherein the device comprisesa packetiser and the first completion time is determined in dependenceon the first encoded frame being received at the packetiser and thesecond completion time is determined in dependence on the second encodedframe being received at the packetiser.
 3. A method as claimed in claim1, wherein the device comprises a packetiser configured to packetise theencoded frames, and wherein the first completion time is determined independence on a first packet for the first encoded frame being outputfrom the packetiser; and the second completion time is determined independence on a first packet for the second encoded frame being outputfrom the packetiser.
 4. A method as claimed in claim 1, wherein thedevice comprises a packetiser configured to packetise the encoded framesinto RTP packets.
 5. A method as claimed in claim 1, further comprisingtransmitting the measure to a device for receiving the media stream. 6.A method as claimed in claim 1, further comprising: receiving a measureof delay in a network; and adjusting a transmission bandwidth for themedia stream in dependence on the measure of delay and the estimatedprocessor load.
 7. A method as claimed in claim 1, further comprising,at a receiving device: forming a measure of delay in a network;receiving the measure of processor load via the network; and adjustingthe measure of delay in dependence on the measure of processor load. 8.A method as claimed in claim 6, wherein the measure of network delay isformed by: determining a third time period between receiving afirst-received packet for the first media frame and receiving afirst-received packet for the second media frame at a receiving device,wherein the packet for the first media frame comprises the firsttimestamp and the packet for the second media frame comprises the secondtimestamp; determining the first time period between the first timestampand the second timestamp; and forming the measure of delay in dependenceon the difference between the third and first time periods.
 9. A methodas claimed in claim 6, wherein the measure of network delay is formedby: determining a third time period between receiving the first mediaframe and receiving the second media frame, wherein the first mediaframe comprises the first timestamp and the second media frame comprisesthe second timestamp; determining the first time period between thefirst timestamp and the second timestamp; and forming the measure ofdelay in dependence on the difference between the third and first timeperiods.
 10. A method as claimed in claim 1, further comprisingdetermining that the processor is overloaded if the second time periodis greater than the first time period.
 11. A method as claimed in claim10, further comprising, in response to determining that the processor isoverloaded, changing parameters for encoding the media frames so as tocause a reduction in the quality of the frames to be outputted by theencoder.
 12. A device for transmitting a media stream, the devicecomprising: an encoder configured to encoding a first media frame and asecond media frame; a processor capable of executing instructions forthe encoder; and a controller configured to: determine a first timeperiod between a first timestamp associated with the first media frameand a second timestamp associated with the second media frame; determinea second time period between a first completion time representingcompletion of the encoding of the first media frame and a secondcompletion time representing completion of the encoding of the secondmedia frame; and form a measure of processor load in dependence on adifference between the first and second time periods.
 13. A device asclaimed in claim 12 further comprising a packetiser configured topacketise the encoded frames, wherein the controller is configured todetermine the first completion time in dependence on the first encodedframe being received at the packetiser and determine the secondcompletion time in dependence on the second encoded frame being receivedat the packetiser.
 14. A device as claimed in claim 12 furthercomprising a packetiser configured to packetise the encoded frames,wherein the controller is configured to determine the first completiontime in dependence on a first packet for the first encoded frame beingoutput from the packetiser and determine the second completion time independence on a first packet for the second encoded frame being outputfrom the packetiser.
 15. A device as claimed in claim 12 furthercomprising a packetiser configured to packetise the encoded frames intoRTP packets.
 16. A device as claimed in any of claims 12, wherein thecontroller is configured to determine that the processor is overloadedif the second time period is greater than the first time period.
 17. Adevice as claimed in claim 16, wherein, in response to determining thatthe processor is overloaded, the controller is configured to changeparameters for encoding the media frames so as to cause a reduction inthe quality of the frames to be outputted by the encoder.
 18. A deviceas claimed in claim 12 further comprising a transceiver configured totransmit the measure to a device for receiving the media stream.
 19. Adevice as claimed in claim 18, wherein: the transceiver is configured toreceive a measure of delay in the network; and the controller isconfigured to adjust a transmission bandwidth for the media stream independence on the measure of delay and the estimated processor load. 20.A device for receiving a media stream via a network, the devicecomprising: a controller configured to form a measure of delay in thenetwork; and a transceiver configured to receive a measure of processorload from a device transmitting the media stream, wherein the measure ofprocessor load is dependent on a comparison of an input frame rate andan output frame rate at an encoder of said transmitting device; whereinthe controller is configured to adjust the measure of delay independence on the measure of processor load.